Many claims for big data projects suggest that just collecting the data somehow gives business the insight they need. The hype is that companies are ignoring this huge resource which is just waiting to be tapped. Early promoters even described big data as “the new oil”.
There is no doubt that IT systems can provide CIOs with vast amounts of data but not all it is useful. There are several steps between collecting big data and turning it into intelligence or useful insight.
The traditional model is a pyramid which moves from Data – Information– Knowledge – and finally Wisdom.
Business should not look at big data as a resource just waiting to be dug up.
The reality is that the second step – turning data into information – is the most difficult, and often expensive, part of the process.
This means cleaning the data and removing rogue or useless results. It means adding context or even making an inference from the data collected.
Knowing that a temperature sensor is reading 800 degrees Celsius is data. Knowing that the sensor is on the office ceiling is information which provides the knowledge that the building is probably on fire.
Working this process on a single sensor is simple.
But dealing with a large data set is hugely complex and runs the risk of adding subjective spin if not handled carefully. In many cases it is actually ‘gut feelings’ about data which are just as important as the data itself.
There are very few data sets which are clean and ready for processing on collection. It is not like oil which can be fed into a refinery to produce petrol and other products.
The irony is that these supposedly objective and automated systems actually require very human intervention to make them work.
The final stage is turning knowledge into wisdom – or into decisions or actions which the business can take.
This requires a system which understands more than the data – it must understand the limits within which the business, and the world, works.
Successful projects are changing many aspects of how businesses function.