How To Manage Big Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Information Management
12:26 PM
Rajan Chandras
Rajan Chandras

How To Manage Big Data

There's an argument for integrating master data management with big data, but much work needs to be done before that happens.

As the discussion around big data gains momentum and substance, questions are being raised about how master data management (MDM) fits into the picture. From what I've seen, the amalgam of master data management with big data is a lot like the Sherlock Holmes story about the dog that didn't bark--what's noteworthy is that there's nothing of note.

I heard (or rather, didn't hear) something similar at the recent MDM and Data Governance Summit in San Francisco, where I presented as well as was part of a panel on master data management and data governance. The message in a nutshell, is that there's definitely a use case for integrating master data management with big data, but there hasn't been much distance covered yet.

The value proposition for bringing master data management into big data analytics is essentially no different from the standard MDM use case: providing identity (i.e. uniqueness) to entities.

Take the product marketing team that's interested in collecting and collating comments made by consumers across the Internet, in discussion forums, personal blogs and other hard-to-decipher places. The data it collects follows a typical big data pattern:

1. Large in size and can quickly go into hundreds of terabytes and beyond.

2. Semi-structured. (Take the phrase "unstructured data" with a grain of salt, because the contents may be free form, but there's almost always some sort of interesting structure around it that can be leveraged for analysis.)

3. Velocity of data is usually high, coming in fast and furious, which poses a challenge to conventional data extract, transform, and load (ETL) architectures and relational databases.

The product is the central point of intersection of master data management and big data. There's potentially important information in all the consumer feedback, but unless we tie in the free-form comments to our product catalog, what use is all that information?

This requires a classic both-ends-to-the-middle approach--technologies and techniques (like natural language processing) to help make sense of the free-form comments at one end, and an accurate and consistent product hierarchy (the product master) at the other end. MDM is clearly an enabler here.

I've heard similar stories elsewhere. At a recent user group conference for a large data warehousing appliance vendor, a pervasive message was the need for mastering identity to support consumer analytics.

Back at the MDM-DG conference, I had an interesting chat with the founder and CEO of Reltio, a venture capital-funded big data initiative, (don't bother visiting the website since it's in stealth mode right now). He had a sound understanding of the challenges of integrating MDM with big data, having previously led product marketing and strategy at a leading MDM vendor. His thoughts around using MDM to support big data analytics resonated with the emerging theme: MDM helps lay the foundation for big data analytics, but we're in the early stages of defining this integration framework.

Beyond big data, the conference featured the usual (and successful) mix of decision-makers, practitioners, and vendors. For example, an interesting presentation on a data governance approach formulated on the basis of Malcolm Gladwell's tipping point theory (loosely defined as a quick and unexpected change that dramatically transforms a situation) can be leveraged to drive data governance deeper and wider in the organization.

The challenges of data governance are fairly ubiquitous. The most common is a shortage of budget and resources, though not, in a curious twist, particularly driven by a lack of executive-level interest in data governance. The mantra for data governance seems to be "doing more with less".

In the products section, two offerings caught my eye, neither of which are new, but seem to have evolved into mature products worthy of attention.

Orchestra Networks' MDM/DG product offers a pleasant change from the integration-heavy architectures of the dominant MDM platforms out there, e.g. IBM, Informatica, Oracle, etc. Orchestra Networks starts with a facility to model your data and take it from there, which presents a much cleaner and more manageable approach, particularly, I suspect, for departmental and smaller MDM implementations.

Talend continues its foray into open source and the cloud. It's a product range that's expanding and maturing fast, and seems to be at or close to the point (call it the "tipping point") where it merits serious consideration in your MDM strategy, especially if you aren't already locked into the IBM, Oracle, or SAP stack. Keep in mind, of course, that "open source" doesn't always equate to "free."

Rajan Chandras has more than 20 years of experience advising and leading business technology initiatives, with a focus on strategy and information management. Write him at rchandras at gmail dot com.

At this year's InformationWeek 500 Conference, C-level execs will gather to discuss how they're rewriting the old IT rulebook and accelerating business execution. At the St. Regis Monarch Beach, Dana Point, Calif., Sept. 9-11.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
9/19/2012 | 3:40:42 PM
re: How To Manage Big Data
Interesting angle- It is true that a common challenge of data governance is a shortage of resources. We have identified a solution to decrease energy costs while at the same time increasing reliability and efficiency. I would be interested to see what you think:
User Rank: Apprentice
7/6/2012 | 4:28:06 PM
re: How To Manage Big Data
This is interesting there was a point raised here In summary MDM is seen as vital to see the BPM people first mindset be taken to enterprise level? Not only that as relevant data is identified and used at the front end so by default the other Gǣversions of the truthGǥ lurking in the legacy mess can be identified and archived? This is the use of MDM on the outside in approach which is now the focus? Interesting to see IBM now push this message? Whilst I am sure the use of MDM to support big data analytics has its place the ROI for improving operational efficiency may prove to be the better option? This aided by significant lower costs to build such front solutions? However this requires a good build tool that can deliver flexible application that represents the future. Naomi Blooms blog hits the spot
GǣGǪ.how those models can become applications without any code being written or even generatedGǥ.
GǣIf IGm right, youGll want to be on the agile, models-driven, definitional development side of the moat thus createdGǪ..Gǥ
Naomi went on to say later Gǣvendors are in trouble if not there (best) or moving quickly to object-based, metadata-driven, definitional development.Gǥ
This has already been created and proven in the UK and it really does work but that MDM capability will be important for larger companies try to sort out the legacy mess.
How COVID is Changing Technology Futures
Jessica Davis, Senior Editor, Enterprise Apps,  7/23/2020
10 Ways AI Is Transforming Enterprise Software
Cynthia Harvey, Freelance Journalist, InformationWeek,  7/13/2020
IT Career Paths You May Not Have Considered
Lisa Morgan, Freelance Writer,  6/30/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Special Report: Why Performance Testing is Crucial Today
This special report will help enterprises determine what they should expect from performance testing solutions and how to put them to work most efficiently. Get it today!
Flash Poll