by Angela Guess
Robin Harris recently wrote in ZDnet, “For IBM, a world leader in AI, as their Watson project has demonstrated, applying intelligence to storage is a natural. We’re facing a data onslaught like never before. We’ll be generating more data than we have capacity to store once IoT gets rolling. Just as any software problem can be solved by adding a layer of indirection, any analytics problem can be solved by adding a layer of intelligence. Of course, we know a lot more about indirection than we do intelligence. IBM researchers are demoing an intelligent storage system that works something like your brain: It’s easier to remember something important, like a beautiful sunset over the Grand Canyon, than the last time you waited for a traffic light. In Cognitive Storage for Big Data (paywall), IBM researchers Giovanni Cherubini, Jens Jelitto, and Vinodh Venkatesan, of IBM Research-Zurich, describe their prototype system. The key is using machine learning to determine data value.”
Harris goes on, “If you’re processing IoT data sets, the storage system’s AI would ‘know’ what is important about prior data sets and apply those criteria – access frequency, protection level, divergence from norms, time value, etc. – to incoming data. As the system watches human interaction with the data set, it learns what is important to users and tiers, protects and stores data according to user needs. The researchers used a learning algorithm known as the ‘Information Bottleneck’ (IB): ‘…a supervised learning technique that has been used in the closely related context of document classication, where it has been shown to have lower complexity and higher robustness than other learning methods.’ IB, essentially, correlates the information’s metadata values to cognitive relevance values with the goal of preserving the mutual information between the two. The greater the mutual information, the more valuable the data and, hence, the higher the level of protection, access, and so on.”
Photo credit: IBM