Dark Data @ Easter Time!

It’s the time of year when I ask my better half if she prefers milk or dark chocolate. She always seems to be in quandary over this! We can use the same analogy for data and its relationships with cognitive computing, IoT and M2M.

Big data is at the heart of creating an effective cognitive system, but there are important differences in the volume and type of information (see the chocolate theme going on here).  We talk all the time about structured and unstructured data, but I’ve yet to see or hear expert advice on dark data and I’m unsure why!

Dark data information is often log data from equipment or security systems. Businesses often mandate storage of this data but rarely seem to use it, due to the expense of analysis.

For example, machine data from manufacturing systems (i.e. SCADA) stored in these logs may support cognitive system predictions for failure (based on patterns of temperature, stress loads and other measurable conditions).

From an IoT and M2M perspective, a by-product of dark data analysis in these logs can be hugely valuable for security mapping to understand patterns that are unknown.  Note the complexities in this, courtesy of some great work done by Beecham Research on threat mapping (http://bit.ly/1nGykjx).

I’d be interested to hear your views on this and how your organisation is integrating dark data along with traditional pattern analysis in its security management plans. Drop me a line with any comments, look forward to hearing from you!