Machine-generated data is one of the fastest-growing segments of the storage market. Organizations are searching for more efficient ways to store and analyze ever-growing stockpiles of digital information, including medical images, consumer-generated photos and videos, and data from unmanned aircraft flying over warzones.

Cloud services such as Amazon Simple Storage Service (Amazon S3) have arrived to address this need, but they're not ideal for all organizations. It may be cheaper, simpler, and more secure to keep data stored where it's collected, process some of the information locally, and then send only that data that needs additional processing up to the cloud.

That's the ideal behind a big data product offered by storage provider DataDirect Networks (DDN) and Alliance Technology Group's YottaStor division. In July, the pair announced that DDN's Web Object Scaler (WOS) cloud storage appliance would be sold as an integrated system with YottaStor’s big data storage system called YottaDrive. Targeted primarily at government agencies that manage huge volumes of intelligence and surveillance sensor data, the WOS/YottaDrive combo is designed to enable data collaboration between distributed facilities and agencies.

[ Intelligence agency investment firm In-Q-Tel is investing in data analysis. Read more at Intelligence Agency Invests In Big Data Management. ]

As its name implies, YottaDrive has cavernous capacities well-suited to big data applications. It's available in three configurations--2.5, 5, or 10 petabytes--with pricing starting at 8 cents per GB per month. YottaStor says that YottaDrive's cost per gigabyte is 90% lower than what many federal agencies working with the company are paying today.

In a phone interview with InformationWeek, YottaStor chief technical director Bob Carlson said that YottaDrive, combined with DDN's distributed architecture that allows geographically disperse users to search and access data wherever it's physically located, is a good way to bring analytics and filtering to an organization's "operational edge"--a remote location where network bandwidth is often limited.

"If you're going to capture data remotely, how do you allow the enterprise--where your analysts are typically located--to access those hundreds of petabytes of data around the world seamlessly?" Carlson asked rhetorically.

The oil and gas industries, for instance, use sensors to collect large amounts of seismic information as part of their exploration operations. And sensors on drone military aircraft gather vast quantities of data in warzones.

"Machine-generated data isn't collected in a single location," said Carlson. "Because sensors are becoming so powerful and affordable, they're typically pushed to the edge of your operation."

One of the main selling points of the WOS/YottaDrive solution is its ability to push big data analytics to the operational edge. "You push those algorithms that do the filtering, the first-level clustering, out to the data," Carlson said. "Since you've already done a bit of refinement, your algorithms in the cloud don't have to do that processing." He added, "Instead of pushing 100% of your data over your networks, you're pulling only the data you need for final processing."

This approach offers advantages to government agencies such as the U.S. Department of Defense (DOD), which is beginning to implement multiple clouds, each with a different focus. "There's a geospatial analytical cloud, a more traditional text analytical cloud, and a message-intercept cloud," said Carlson.

The era of big data, cloud services, and distributed computing is in its infancy, and Carlson believes many innovations are on their way. Using a baseball metaphor, he said, "We're in the top half of the first inning, no outs." Carlson predicted that the cloud appliance model exemplified by WOS and YottaDrive will be successful because it allows organizations to maintain control of their data.

By comparison, cloud services offered by Amazon and other providers are appealing, but they may not be the best choice for large enterprises. "They want flexibility, and they're going to look for different approaches to take advantage of the cloud computing model. But [they won't] give up control of the data," Carlson said.

InformationWeek Government's GovCloud 2012 is a day-long event where IT professionals in federal, state, and local government will develop a deeper understanding of the options available today. IT leaders in government and other experts will share best practices and their advice on how to make the right choices. Join us for this insightful gathering of government IT executives to hear firsthand about the challenges and opportunities of cloud computing. It happens in Washington, D.C., on Oct. 17.