How to Discover the Hidden Patterns in Big Data


Karlin Lillington of The Irish Times reports, "At IBM’s Almaden research centre in the hills just outside of San Jose, big data is a big part of current research. While much of the work that happens at Almaden is 'blue sky' research that looks far into the future, many of the big data projects, involving collaborative work with current IBM clients, are likely to have more immediate applications." One such project is using text analytics to help accelerate pharmaceutical research: " 'Developing one drug is a 10- to 15-year cycle,' says Ying Chen, a researcher, master inventor and manager in IBM Almaden services research. 'They say developing drugs is like casino gambling. The blind alley is a big problem in this industry. The drug failure rate is around 90 per cent'."

Lillington goes on, "But put the concentrated computing power of Watson – the IBM supercomputer that beat two human opponents in 2011 on the US game show Jeopardy! – to sifting through the text of 20.6 million Medline medical journal articles and 16.2 million patent applications, and the odds of finding the best candidate drugs and therapies could shorten considerably. From this information, IBM has extracted searchable bits of information called 'semantic entities'. These include 12 million chemicals, 7,000 diseases, 22,000 drugs and 130,000 genes."

Read more here.

Image: Courtesy IBM