Big Data Talent War: 10 Analytics Job Trends

IBM announced a significant upgrade of its DB2 database and supporting InfoSphere software on Tuesday, packing it with performance improvements for both transactional and data warehousing operations.

The upgrades answer advances seen across the database market in recent months (and, in some cases, years), but the competitor IBM seems most intent on challenging is database marketshare leader Oracle. Even before last year's HP-Oracle Itanium flap--a legal skirmish that's ongoing--IBM was intent on exploiting discontentment among Oracle customers. The two key appeals are promises of lower cost and an easy transition from Oracle management and administration methods.

Coca Cola Bottling Company made the switch from Oracle to DB2 four years ago, and it figures it has saved $750,000 on license and storage costs over the last five years. Coke Bottling IT exec Andrew Juarez said the latest version of DB2 will help it save even more money. First a bit of detail on the upgrades he plans to exploit.

The DB2 10 and InfoSphere Warehouse 10 upgrades announced today and set for release at the end of the month include five key upgrades: Time Travel Query, Multi-Temperature Data Management, Adaptive Compression, Continuous Data Ingest, and a connector to the Hadoop distributed file system (HDFS). Some of these upgrades will sound familiar, as rivals have done it first. But IBM brings some new twists and the advantage of serving both transactional and data warehousing needs.

[ Want the latest on database trends and satisfaction? Read Research: State of Database Technology 2012. ]

The Time Travel Query feature is akin to temporal-data-management advances made last year by Teradata. But IBM says it's unique in addressing future dates as well as past dates and in applying time-based querying to transactional environments. A travel agency, for example, could spot scheduling conflicts whereby a customer has booked a car in Rome when his flight itinerary shows he's supposed to be in Los Angeles.

"This also comes up in insurance when you need to know policy details that were in force in the past, or in banking, when you're doing financial analysis and a past error is discovered and balances have to be recalculated as of a certain date," said Bernard Spang, IBM's director of strategy and marketing for database software and systems, in an interview with InformationWeek.

The Adaptive Compression feature of DB2 finds repetitive data across the database page and table levels, replacing that data with a small, storage-saving key. As a result, DB2 10 is said free up significant storage space, depending on the nature of the data. The state-of-the-industry in data warehousing in recent years has been 4X compression, but with vendors including EMC Greenplum, HP Vertica, Oracle, and Teradata exploiting various forms of columnar compression, standards have changed.

Adaptive Compression delivers the equivalent of 7X to 8X compression, according to Spang, and that also brings efficiencies in querying, with an approximate 10% improvement in response times over the previous DB2 release, he said. Oracle claims peak compression rates of 10X with the hybrid columnar compression, which is offered with its Exadata Database Machine.

Multi-Temperature Data Management also strikes familiar cords struck in Teradata releases over the last two years, but again, that vendor addresses data warehousing alone. The feature automatically moves hot (frequently accessed) or cold (infrequently accessed) data to an appropriate--faster or slower--storage medium, based on policies set by the customer. Benefits include improved performance and reduced cost. In DB2's case, these advantages are equally important in transactional settings, as it reduces storage costs and can improve performance.

As for the Continuous Data Ingest tool, here, too, other vendors have been there and done that. Continuous Ingest is vital if you want real-time analysis, as it gets you out of overnight big-batch loading in favor of continuous trickle feeding. Oracle, for one, got there a couple of years ago with its acquisition and integration of software from GoldenGate, which specialized in fast-data-loading techniques.

Finally, HDFS connectors are the check mark du jour. Any database that hopes to tap into big data will start with Hadoop connectivity.

Despite all the buzz about big data in IBM's press release, customers such as Coca Cola Bottling Company are most interested in fundamentals like better compression. With its initial move from Oracle to DB2 back in 2008, Juarez says Coke Bottling saw a 40% storage savings, taking its SAP system database from a terabyte down to 590 gigabytes. Subsequent DB2 upgrades delivered another 17% reduction in storage requirements. And based on tests of the beta version of DB2 10, applying Adaptive Compression to 160 of the largest tables among the tens of thousands in the company's SAP system deployment, Juarez said he expects to see an additional 20% reduction in storage.

"After four years on DB2, my database is back to just slightly over a terabyte, so we haven't had to add storage," Juarez said. Compression benefits starts off with storage savings, he said, "but it also shows up in performance because with every read you're bringing back more records into memory."

As for that warm and fuzzy feeling developers and admins get when working with familiar products, Juarez said it didn't take long to switch gears with the transition from Oracle to DB2 back in 2008.

"We had reservations because we were so familiar with Oracle, but we spent two and a half months doing a prototype, and we became very comfortable with it," Juarez said.

Or course Oracle Database hasn't stood still all these years, adding compression advances, engineered systems, and plenty of other bells and whistles and related software, but Juarez said he's not looking to go back.

When picking endpoint protection software, step one is to ask users what they think. Also in the new, all-digital Security Software: Listen Up! issue of InformationWeek: CIO Chad Fulgham gives us an exclusive look at the agency's new case management system, Sentinel; and a look at how LTE changes mobility. (Free registration required.)