What's Next for IBM Watson? Big Data

Inside Watson, IBM's Jeopardy Computer

At the IBM Software Innovate2011 conference, IBM engineers this week revealed some clues to the company's plans for Watson, the company's artificial intelligence research project that saw significant exposure during its run on the TV game show Jeopardy, where it beat human contestants.

IBM has a long history of pitting humans against machines, hoping that the efforts will improve computing performance and lead to designs where machines can emulate human intelligence. Prior to Watson, IBM took on chess grandmasters with the company's "Deep Blue," a system designed to evaluate tens of thousands of potential chess moves and develop a winning strategy.

However, Watson is very different from Deep Blue and faced different development challenges. For example, Watson needed to understand the nuances of human language and communicate using the same medium. Understanding language and processing data from multiple sources concurrently are some of the differences from Deep Blue that may make Watson a powerhouse of collaboration in the world of business intelligence, where artificial intelligence can augment research, instead of competing with humans.

During one of the many presentations about Watson at the Innovate2011 conference, IBM researcher Aditya Kalanpu said, "With chess there are only so many moves you can make, but language presented new challenges. This is very hard for a machine to do, but we've taken a big step with Watson. We're a long way from full language understanding, but this is a major move forwards."

During a group interview, Walker Royce, IBM's chief software economist, pointed out that Watson has the potential to change how humans work with machines. "Watson's analytical capabilities and its ability to interact with humans could make it ideal as a member of a team, collaborating on solutions to complex problems, especially in the medical and legal fields," he said.

Of course, medical and legal are not the only potential areas that Watson can help out in. A growing market of Big Data analytics seems to be a natural fit for Watson's capabilities. If successfully commercialized, Watson could find new trends and previously undiscovered relationships between large data sets, which could bring additional value to the massive data stores that many enterprises are forced to keep today.

For example, if Watson does indeed evolve as IBM engineers hope, Watson should be able to bring customers some very esoteric capabilities, such as market analytics that could compare customer histories with global trends, backed by blog research and current news, which could transform the investment and stock markets.

However, challenges remain. The Watson team spent four years designing a system capable of understanding language, syntax, and simile on a human level. Doing so required massive processing power, on the level of the world's top supercomputers. That meant going back to the drawing board and coming up with new hardware and software designs, resulting in Watson being built using massive parallel processing systems. The configuration used for Jeopardy used 2,800 processing cores, a level of processing that was needed to generate answers in seconds. Kalanpu said using a standard PC, the Watson software would take about two hours to answer a question.

Watson works by breaking down questions into component parts and then searching for possible answers, rating each one against the evidence for or against the proposition and based on a variety of data metrics. One of the key differentiators of IBM's software is a technology dubbed DeepQA, which can mine massive amounts of both structured and unstructured data for results.

What's more, IBM's researchers let the machine learn from choosing right and wrong answers, allowing them to tune the algorithms to make them more accurate. With Watson being fine-tuned for commercial use, IBM's researchers hope they can accurately setup specific data sets in just a few hours.

Time will tell if Watson will transform research, analytics, and human-machine interaction. But one thing is sure, machines are getting smarter and will eventually collaborate with humans, with the machines taking on more human characteristics. Perhaps one day you'll take your coffee breaks with Watson.

Employees have more ways to communicate than ever, but until the mishmash of tools gets integrated, productivity will suffer. Also in the new, all-digital issue of InformationWeek: A buyer's guide to enterprise social networking. Download it now. (Free registration required.)