Microsoft won't be selling its own data warehouse appliances. But by buying appliance vendor DatAllegro, in a deal announced July 24, it will be able to integrate support for shared-nothing, massively parallel processing (MPP) architecture into Microsoft SQL Server. Translation: With the help of hardware and distribution partners, Microsoft intends to leapfrog Oracle and step up to high-scale data warehouse deployments.

"We're significantly raising the bar in terms of the scale that our customers will be able to reach with [the upcoming release of] SQL Server 2008 into the tens of terabytes," said Fausto Ybarra, director of SQL Server product management. "The DatAllegro acquisition will accelerate our data warehousing roadmap and quickly take us into high-scale deployments in the hundreds of terabytes."

Data warehouse appliances have been a hot hand for more than three years as more warehouses scale up into the double- and triple-digit terabyte range. Most appliances exploit shared-nothing, MPP architecture. Shared-nothing means that data nodes are independent, with dedicated memory and storage, while MPP spreads query load across the many processors available in commodity multicore processor servers. The result is high performance without using expensive, high-powered symmetric multiprocessor (SMP) servers. Appliances also are easier to deploy and maintain than conventional warehouses, which have to be tuned, optimized and, lately, clustered to perform in large-scale deployments.

Teradata pioneered shared-nothing, MPP more than a decade ago, but today there are more than a dozen vendors exploiting the technology. So why did Microsoft choose DatAllegro, a company with a high profile but few customer references to its credit?

"From a technical perspective, DatAllegro is based on an open architecture using industry-standard hardware, which makes it easier to integrate into existing SQL Server infrastructure," said Ybarra. "We considered other vendors, but we concluded that DatAllegro had the best fit with our architecture, and it will be relatively straightforward to integrate the technology into SQL Server."

Rather than selling appliances directly, Ybarra says Microsoft will work with hardware vendors including Dell, HP, Bull, EMC and Cisco as well as distributors to bring preconfigured packages to the market. Microsoft says it will retain DatAllegro's management and development teams as well as its offices in Aliso Viejo, CA, but it did not disclose the terms of the acquisition nor would it comment on how long it would take to integrate DatAllegro's software or how the switch to SQL Server would impact cost; DatAllegro currently employs the open-source Ingres database and the appliance runs on the Linux operating system. Ybarra said a product road map and packaging plans will be detailed at Microsoft's BI Conference in October.

When will it make sense for Microsoft SQL Server customers to add an appliance? That will depend on the volume of data, the number of users and the complexity of the queries. "I've seen customers who have a relatively small, 10-terabyte warehouse, but they have 1,000 users who are continuously submitting ad-hoc queries, so that's a high-end scenario," Ybarra explained. "Another customer might have a data warehouse that's three times as large but with only a handful of users, so that may not be a high-end scenario."

The DatAllegro acquisition marks the first sign of consolidation in what has been a fast-growing market, with many new startups seeking out the same would-be customers, such as telecommunications companies with billions of call data records and big retailers, banks and insurance companies studying up-sell and cross-sell opportunities as well as churn, fraud and risk trends.

"We started to see a lot of startups, and the market was getting overcrowded," said Stewart Frost, CEO of DatAllegro and soon-to-be head of Microsoft’s global Data Warehouse Center of Excellence. "It's always best to be one of the first to be part of the consolidation in a market, and you don't want to miss out. The Microsoft deal made sense for us, because there's a clean technology fit and no market overlap."

Given that Teradata and IBM already offer data warehouse appliances aligned with their databases, the DatAllegro acquisition puts particular pressure on Oracle, which only offers partner appliance configurations built on SMP servers. Oracle declined to comment on the deal, but for to read comments from other vendors, read "Competitors React to the Microsoft-DatAllegro deal."

"Shared-nothing, MPP architecture is absolutely the right way to do data warehousing," said independent information management analyst Curt Monash. "Collectively the data warehouse appliance vendors have been kicking butt, so Oracle had to develop or acquire this technology anyway. The Microsoft acquisition of DatAllegro just adds to the urgency of doing so."

For the independent data warehouse appliance vendors remaining, today's news means there is one less would-be acquirer and the prospect of much tougher scrutiny for future rounds of venture capital funding. In short, they'll have to get acquired, differentiate or die (read Mark Madsen's blog on "What the Microsoft-DatAllegro Deal Means to Customers, Vendors and BI").