Data Debate - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

02:07 PM
Connect Directly

Data Debate

Security or privacy? Darpa's Total Information Awareness program tests the boundaries

More details about one of the world's most ambitious--and controversial--database projects will be released this week when the Defense Advanced Research Projects Agency issues a report to Congress on its Total Information Awareness program. The initiative has been a point of contention for privacy groups and others worried that Darpa or other government agencies will collect personal information on U.S. citizens in a national security effort to thwart terrorists through sophisticated data-analysis and collaboration techniques. If Darpa can overcome the privacy concerns, some experts say, its bleeding-edge prototype system just might work. "We're at the cusp of actually being able to do it," says data-warehousing consultant Bill Inmon, citing advances in storage, database, and microprocessor technologies, accompanied by continuing declines in hardware costs, that make such a large-scale undertaking conceivable.

Total Information Awareness represents a data-management challenge as bold as it is controversial. The goal, as Darpa's Information Awareness Office (headed by John Poindexter, former national security adviser in the Reagan administration) envisions it, is to sift through vast quantities of data contained in government and business databases, detect suspicious patterns of activity, identify the shady characters behind those actions, then find them before they can do any harm. The five-year research and development project would test the limits of database integration and scalability and require breakthroughs in language translation, pattern matching, and agency-to-agency collaboration. "It is anticipated this will require revolutionary new technology," Darpa said last year in a document soliciting bids from potential suppliers.

How would it work? According to public comments made by Darpa officials and information on the agency's Web site, the goal is to create an architecture capable of building a huge database that gets populated, with the help of automated processes, from existing databases. A database crawler would be used to recognize the data structures of source databases, facilitating the flow of data. Data-mining algorithms would be run against the data collected and models generated in an attempt to predict terrorist behavior. Collaboration tools would link experts together for quick action when a terrorist's "signature" is suspected.

"One goal is to develop ways of treating the worldwide, distributed legacy databases as if they were one centralized database," Poindexter said last year. Total Information Awareness was launched in fiscal 2003 with $10 million in initial funding, but related projects predate the terrorist attacks of Sept. 11, 2001. Darpa will spend an estimated $240 million on the combined projects from 2001 through 2003. Prototypes of the system would be turned over to other Department of Defense agencies for adoption.

Illustration by Michael Morgenstern

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 3
Comment  | 
Print  | 
More Insights
2020 State of DevOps Report
2020 State of DevOps Report
Download this report today to learn more about the key tools and technologies being utilized, and how organizations deal with the cultural and process changes that DevOps brings. The report also examines the barriers organizations face, as well as the rewards from DevOps including faster application delivery, higher quality products, and quicker recovery from errors in production.
The State of Chatbots: Pandemic Edition
Jessica Davis, Senior Editor, Enterprise Apps,  9/10/2020
Deloitte on Cloud, the Edge, and Enterprise Expectations
Joao-Pierre S. Ruth, Senior Writer,  9/14/2020
Data Science: How the Pandemic Has Affected 10 Popular Jobs
Cynthia Harvey, Freelance Journalist, InformationWeek,  9/9/2020
Register for InformationWeek Newsletters
Current Issue
IT Automation Transforms Network Management
In this special report we will examine the layers of automation and orchestration in IT operations, and how they can provide high availability and greater scale for modern applications and business demands.
White Papers
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Sponsored Video
Flash Poll