Big Data Predictions For 2016

The practice of arguing over the validity of technology benchmark tests may be only about a day older than the existence of benchmark tests. It's a longstanding IT industry tradition to try to prove whose product is better with some kind of showdown and use the results to win customers. But organizations often disagree about what should be tested, who should test it, and plenty of other factors of testing in an effort to ensure the tests are fair (and that their own products come out on top and don't underperform).

One such battle has been brewing in big data recently between Informatica and Talend. It all started with a Talend-commissioned benchmark test conducted by MCG Global Services in October 2015 that said Talend Big Data Integration offered about 10 times faster performance than Informatica Big Data Edition.

Not surprisingly, Informatica objected to the validity of these results, saying the benchmark was not independent because it was commissioned by the winner, and Informatica wasn't consulted. Informatica also said in a blog post that the benchmark test pitted its two-year-old product against Talend's month-old product.

It so happened that Talend's product was released a month before the benchmark was conducted, and Informatica's new product was released a month after the benchmark test. Plus, Informatica said in its blog post, the MCG Global Services test didn't follow the Transaction Processing Performance Council (TPC) standard for vendor-neutral benchmark tests.

[Looking for more information about Informatica's Big Data Management Platform release? Read Informatica Big Data Management Platform Aims for Next-Level Integration.]

Talend's CEO Mike Tuchen told InformationWeek in an interview that some of Informatica's objections to the benchmark test did indeed have merit. So how is he proposing to level the playing field?

Talend has issued a public challenge to Informatica to create a new benchmark test. In a blog post last week, Talend CEO wrote: "Let's use a certified configuration agreeable to both of us, a mutually agreed benchmark, and allow both companies to give technical guidance to the company running the benchmark."

InformationWeek recently spoke with the CEOs of both companies to discuss the benchmarks. Here's what each company had to say:

Tuchen said that Talend pursued the original benchmark test after it had heard glowing reviews from its customers following a switch it made to Apache Spark from MapReduce about 18 months ago.

"This turned out to be a terrific bet," Tuchen said. "We had a number of customers running their own side-by-side comparisons. They said as far as they were concerned, we were the fastest. Having our customers tell us that we were the fastest in the world was exciting."

So Talend hired a consulting firm to run a benchmark looking at Talend's product versus Informatica's product and published the results in October. No other vendors were evaluated as part of this benchmark, Tuchen said.

Informatica CEO Anil Chakravarthy told InformationWeek in an interview that the company would evaluate the challenge from Talend based on whether it would serve the business needs of customers, and not solely for the sake of an experiment.

"If we think it is something that will give customers business value then we are happy to do it," he said.

Informatica noted in an email to InformationWeek that the benchmark test commissioned by Talend in October focused on differences in the performance of MapReduce versus Spark, two streaming technologies for big data. While the old version of Informatica that was tested in the benchmark used MapReduce, the new version uses Informatica Blaze Engine on YARN instead. Informatica said the latter has been shown to run faster than Spark and MapReduce for batch ETL transaction processing.

Informatica also said that the data sets used in the benchmark were too small to be considered big data.

CEO Chakravarthy is new to the role of CEO at Informatica, having been appointed as acting CEO in August 2015, after the company went private in an acquisition by private equity firms, with additional investments from Microsoft and Salesforce. That appointment was made permanent earlier this month. Chakravarthy served as chief product officer at Informatica before assuming the CEO job.