Information overload has become a significant challenge for the U.S. military and will require new analysis software and a Google-style cloud infrastructure to manage massive data sets, a U.S. defense advisory group report finds.

The December 2008 report, "Data Analysis Challenges," was initially withheld from the public. It was obtained by the Federation of American Scientists' Project on Government Secrecy through a Freedom of Information Act request.

The report, written by JASON, a group that provides advice to the Department of Defense (DoD) through the non-profit MITRE Corporation, says that the massive amount of sensor and imagery data being gathered is becoming increasingly difficult to store, analyze, and integrate into defense systems.

For example, a DoD surveillance system called Constant Hawk typically produces 10's to 100's of Terabytes of data over a period of a few hours. For that information to be useful, it has to be stored, analyzed, and distributed quickly.

The report, however, cites concerns voiced by members of the defense and intelligence communities that much of the surveillance data gathered isn't made useful.

"Seventy percent of the data we collect is falling on the floor," MIT defense research scientist Pete Rustan said, according to the report.

And the problem is likely to get worse. "As the sensors associated with the various surveillance missions improve, the data volumes are increasing with a projection that sensor data volume could potentially increase to the level of Yottabytes (10^24 Bytes) by 2015," the report says.

Jonathan B. Spira, CEO and chief analyst at research consultancy Basex, author of the forthcoming book Overload!, and organizer of Information Overload Awareness Day on August 12, says information overload is a real problem in the workplace, in government and in the military.

"We've seen on the military side, many instances where information overload can create a whole new kind of fog [of war]," he said.

Information overload costs the U.S. economy $900 billion per year, according to Basex.

The JASON report discounts some of the more extreme projections about data volume growth and recommends that the DoD deploy infrastructure similar to that used by Google, Microsoft, and Yahoo. It also sees military applications for the Hive language used by Facebook for data warehousing.

The major problem the DoD faces will be in the area of automated information analysis. "The notion of fully automated analysis is today at best a distant reality, and for this reason, it is critical to invest in research to promote algorithmic advances," the report says. "One way to effectively engage the relevant research communities is through the use of grand challenges in the area of data analysis."

Spira sees information overload as a broader problem, one that won't vanish with the development of improved automated information analysis technology. He described a cybersecurity conference at a Maxwell Airforce Base, where military brass had gathered to discuss cyber threats. Emerging from the talk, the generals found they had no e-mail, he said. It turned out that the base's e-mail system had been taken down, not by a cyber attack, but by an e-mail about a card game that got forwarded and, through too many reply-alls, multiplied until over a million messages overloaded the e-mail servers.

"We need to address a lot of different aspects of data and information overload, not just things that sound sexy," said Spira.