EMC's Pivotal Plan: Brilliant Or Crazy?

Pivotal, the company spun out of EMC and VMware on April 1, formally announced its launch on Wednesday, unveiling a truly grand plan. In fact, if you consider all the components and ambitions, you might conclude the plan is either brilliant or crazy.

The need for Pivotal's promised "next-generation enterprise computing platform" starts with the premise that there has been a fundamental shift in IT, with Internet consumer giants like Google, Facebook and Amazon pioneering large-scale data-management approaches and rapid application-development capabilities supported by unprecedented levels of automation.

These approaches have yet to effectively reach the vast majority of companies and government agencies still mostly dependent on client-server applications built on relational databases and even mainframes, said Paul Maritz, Pivotal's CEO. What's needed, he said, is a next-era platform that can integrate with those legacy technologies.

[ Want more on Pivotal's technology and business plans? Read EMC, VMware Prep Big Data Analytics Spinoff. ]

"We have to bring together the ability to ingest large amounts of data, develop deep understanding and then do something, in real time, based on that insight," Maritz said, kicking off the launch event in San Francisco. "These kinds of applications are emerging across a wide variety of industries, but there's no easy, cost-effective way to do it on existing infrastructure and relational databases, so that's a strong hint that there's a new platform waiting to be born."

Plans for Pivotal were first announced in December, and last month Maritz detailed a business planthat foresees $1 billion in revenue by 2017. More details emerged on Wednesday along with the surprise announcement that General Electric is taking a $105 million stake in the company.

GE plans to take advantage of Pivotal's platform to power its efforts to build an industrial Internet, GE CEO Jeffrey Immelt said in a prerecorded video. The manufacturer will use Pivotal's platform to capture and analyze data from smart devices such as turbines, aircraft engines, locomotives and medical equipment, he said.

"It's about smart machines, big data and analytics, and mobile workforces, and those coming together for airlines, utilities, oil and gas companies, and healthcare providers to provide great applications like no unplanned downtime, asset optimization and enterprise optimization," Immelt said.

The three key components of the Pivotal platform, which will see its first release in the fourth quarter, are an abstraction layer for cloud computing, big data infrastructure with associated real-time analytics capabilities, and an agile application development environment. The company has a head start in each of these areas, said executives, thanks to software, communities and people contributed by both EMC and VMware. The company is launching with 1,250 employees moving over from EMC and VMware.

Amazon, Microsoft and others have built out cloud services for corporations and government agencies, but Pivotal said a key need yet to be addressed is providing a layer of abstraction affording cloud users a degree of independence from any one provider. The idea is to automate the provisioning of public and private, with the public options including infrastructure-as-a-service (IaaS) providers such as Amazon, Microsoft and others.

"Enterprises aren't all going to want to build [or work on] consumer, Internet-scale clouds, so we're going to have to offer them choice as to which cloud they deploy their applications on," Maritz said. "We need to offer enterprises a way to automate and scale with portability across clouds, which are the modern equivalent of hardware."

Pivotal said the "cloud fabric" that will provide this capability will be based on Cloud Foundry platform-as-a-service (PaaS) software and expertise from VMware.

Pivotal's big data and analytics capabilities blend Hadoop with EMC's Greenplum database, VMware's GemFire in-memory caching technology and the HAWQ (Hadoop With Query) SQL querying capabilities recently announced as part of a coming Pivotal HD Hadoop release.

Pivotal executives have said that they're "all in on Hadoop," but with HAWQ and GemFire, they're addressing the two biggest weaknesses of the platform, which are its currently slow and limited SQL query capabilities (as supported by Hive) and the general lack of performance.

"We announced the massively parallel processing query service in Pivotal HD through HAWQ, but you can expect advances from Pivotal in real-time database processing," said Scott Yara, Pivotal's senior VP, products and platform.

The third leg of Pivotal's strategy will be built on the application development expertise of Pivotal Labs, contributed by EMC, and VMware's SpringSource unit. Spring provides a suite of software products for building, running and managing Java Web applications, and it also provides training on the open source Spring Framework for Java application development. Yara noted that three to four million developers rely on Spring for application development.

"You have to be able to easily move from development to test to production without a process-heavy approach ... and you have to provide an auto-deploy and scaling mechanism as part of the application fabric," Yara said.

With independence, Pivotal will be free to work with IaaS providers, such as Amazon and Microsoft, that might not have partnered with EMC or VMware, Maritz said. Independence helped VMware, where Maritz was the longtime CEO, move out from under EMC's wing, growing from a $635 million acquisition in 2004 into a separate company now valued at $32 billion.

Parent firms EMC and VMware are clearly hoping for a repeat growth story, but in going after so many aspects of enterprise computing, Pivotal is likely to make plenty of enemies. Fostering cloud independence, for example, isn't likely to endear Amazon or other cloud providers. And in the Hadoop arena, Pivotal's go-in-alone plans for HAWQ SQL querying and real-time performance with GemFire have competitors questioning Pivotal's use of Hadoop.

"What appears to be missing from [Pivotal's] strategy is real participation in the Apache Hadoop community, and more specifically, development in the open source projects like [the Hadoop Distributed File System] that are the foundation of their solution," said Cloudera CEO Mike Olson in an statement emailed to InformationWeek.

EMC has not made much headway as a Hadoop distributor through Greenplum, and, meanwhile, Cloudera and others are also working on fixes for Hadoop's flaws. Cloudera, for one, developed Impala for SQL querying on Hadoop, a project that is currently in beta.

Pivotal faces the complexity of coordinating multiple EMC- and VMware-derived products that weren't developed or designed to work together. The firm must also satisfy the needs of the divergent constituencies responsible for providing compute capacity, data-management and analytics, and application-development capabilities. Will inroads in any one of these areas naturally translate into opportunities in another? Or will professionals in each camp stick with favored, best-of-breed services and providers?

"[Pivotal's] success will hinge on their ability to compete flawlessly across multiple fronts against very large and established vendors such as Amazon, Microsoft, IBM and Oracle," said Shaun Connolly, VP of corporate strategy at Hadoop software distributor Hortonworks, in a statement. "Rationalizing and enhancing technologies across application runtimes, analytic tools, data warehouse technologies, development frameworks and more will certainly be a complicated endeavor."

In a Gartner research note published Wednesday, the analyst firm lauded Pivotal's vision but also pointed to gaps in its portfolio including a lack of data-, application- and cloud-integration technologies and mobile and social capabilities.

"Considerable functionality will have to be developed, acquired and integrated into the platform before it can claim victory," Gartner wrote.

Will Pivotal's reach prove to be beyond its grasp? That's a question only time can answer, but there's no doubt Pivotal's vision will help spark more reinvention of corporate computing approaches.