Big Data Talent War: 10 Analytics Job Trends

When astronomers turn on the Square Kilometer Array (SKA), expected in 2024, the massive radio telescope will begin capturing as much as one exabyte of raw data per day. That's about twice the amount of data currently generated on the World Wide Web each day, but that load is spread across the more than 20,000 autonomous computer systems that power the Internet.

What will it take to build a single computer system that can read, store, and analyze all that data? The Netherlands Institute for Radio Astronomy, known as Astron, and IBM Research announced Monday a five-year, $42 million "DOME" collaborative to develop an architecture for the extremely fast, but low-power platform that will be required.

The SKA, which will become the world's largest and most sensitive radio telescope, is to be developed by an international consortium supported by 20 countries, including Australia, Canada, Chile, France, Russia, the United Kingdom, and the United States. The location has yet to be finalized, but the leading options are in Australia, New Zealand, and South Africa. Construction of the millions of antennas required to complete the project is expected to begin in 2017, so the DOME project will conclude in time to yield a workable architecture.

[ Want more on space exploration? Read NASA Releases Atlas Of Entire Sky. ]

"We understand that the technology available today will evolve, so to use a hockey term, you have to skate to where the puck will be five to eight years from now," explained Ronald Luijten, manager of data center optimization at IBM Research, Zurich, in an interview with InformationWeek.

Current technologies are either too power-hungry, too costly, or too slow and small-scale to handle the SKA challenge. The compute power required will equal millions of PCs, and current data center technologies won't do, as they expend as much as 98% of required power moving data and only 2% doing the actual compute work. Researchers believe 3-D chips, which are starting to be used in a simple form in small-scale electronic devices such as iPhones, would be a first step to delivering the required power without extreme energy demands.

"By putting tens of layers together on a single chip, the distance of connection goes from tens of centimeters on a circuit board to less than a millimeter," said Luijten. "By putting 3-D memory stacks directly on our CPUs, we can shorten the memory bus and perhaps even rid one level of cache, which will also save energy."

Storage presents another major challenge, as the SKA will present an exascale daily data load and long-term storage demands of 300 to 1,500 petabytes per year. Current DRAM is fast enough to handle expected speed requirements, but it's too expensive and power hungry. Flash memory is more affordable than DRAM, but it doesn't meet the speed or durability requirements.

The leading candidate for short-term storage is phase-change memory (PCM), an emerging solid-state technology that uses electrical signals to change the resistance characteristics of special metal alloys. IBM's researchers say PCM is 100 times faster than flash, supports high capacities, and does not lose data when turned off. It's also durable, handling 10 million write cycles, compared to 30,000 for current enterprise-class flash and 3,000 for consumer-class flash.

PCM is too expensive for long-term storage needs and spinning disks are too power hungry, so the leading candidate here is tape. Despite its antiquated image, tape is still the low-cost storage king, and existing roadmaps for the technology foresee 100-terabyte cartridges within the next 10 years. DOME research will study how SKA data can be stored on tape while still supporting active-archive streaming data reading.

Connecting the data collection, compute, and storage aspects of the platform presents a third major power and speed challenge, and the SKA will be spread over a large area. Nanophotonics, an emerging technology that uses light to transmit signals across silicon fibres, is a leading candidate.

"We're studying whether we can sample radio signals directly using photonics rather than electronic methods, so we remove an intermediate step and do everything with [faster and more energy-efficient] photonics," Luijten said.

The DOME collaboration is being financed in large part by the Dutch government, which is promoting the development of the knowledge economy in the Province of Drenthe. A DOME lab to be opened by Astron and IBM in the town of Dwingeloo, the Netherlands, is expected to employ 30 to 50 researchers initially and eventually create 200 to 300 supporting jobs.

The technologies developed through the DOME collaboration will not only benefit the SKA; it's hoped they will contribute to more powerful and energy-efficient commercial data centers in the decades ahead.

When picking endpoint protection software, step one is to ask users what they think. Also in the new, all-digital Security Software: Listen Up! issue of InformationWeek: CIO Chad Fulgham gives us an exclusive look at the agency's new case management system, Sentinel; and a look at how LTE changes mobility. (Free registration required.)