Concurrent is in the interesting position of watching from the sidelines as different vendors and projects vie for adoption and supremacy in the big data arena. Concurrent's task is to make sure that Cascading and its app-performance-management system, Driven, can work with every option that's viable and at least somewhat popular.
Here are some other interesting observations shared during my interview with Concurrent execs:
Right now, people just want something that's faster than MapReduce. Do they care what that is? Not really, as long as it's faster. Do they want to have to learn yet another API for parallel data computation -- Spark, Tez, whatever? Not really, but if it's faster than MapReduce, they'll try it. -- Chris Wensel, founder and CTO, Concurrent
The pragmatic, real world, it's not going to be all or nothing. It's not going to be all MapReduce or all Tez or all Spark or all Storm. There's always going to be some newfangled computation engine that somebody's going to get excited about. But the reality is that you have to solve business problems today. So it's going to be all of the above, depending on the size of data, the latency that the customer is willing to tollerate, and the reliability that you're willing to tollerate. -- Gary Nakamura, CEO, Concurrent