Big data is big news these days. But most organisations just end up hoarding vast reams of data, leaving them with a massive repository of unstructured – or “dark” – data that is of little use to anyone.
Given the potential benefits of big data, it’s crucial that we find better ways to gather, store and analyse data in order to make the most of it.
Stories of big data successes have triggered significant investments in big data initiatives. This has prompted many organisations to gather significant volumes of external and internal data into so-called “data lakes”. These are repositories that contain data in any format, whether structured, like databases, or unstructured, like emails or audio and video.
As a result, the growth in the amount of data being generated, collected and stored continues at an exponential rate.
But according to a recent IBM study, more than 80% of all data is inactive, unmanaged, often unstructured, lacking meaningful metadata, and even unknown to the organisation. The proportion of this dark data is expected to reach 93% by 2020.
For example, data generated from vehicle on-board devices can be expected to reach 350MB of data every second. Where does all this data go and who is using it?
Organisations can also generate significant internal data. For example, a recent study found that a company with 1,500 employees had around 2.5 million spreadsheets, each of which were only used by 12 people on average.
What’s more, there is evidence of a variety of unstructured data such as document versions, project notes and emails that is left behind from organisational processes and subsequently sits dormant in data servers.
Lessons learnt from years of research in information system use have shown that the assumption that “more is better” when it comes to data is unfounded.
Even in traditional IT projects that follow carefully crafted analysis and design life cycles, the misalignment between perceived and actual value has been a notoriously difficult problem, often leading to poor returns on investment.
In big data projects, the data can often be externally sourced with little or no knowledge of its schemata, quality or expected utility. Thus the risk of making investments that will not deliver is greatly heightened.
The old adage of “use it or lose it” is by no means obsolete, and brings attention back to the purpose of how we use big data. Organisations may retain data for a variety of reasons, including data retention regulations, but perceived future value is typically the main reason.
Although storage is relatively cheap, given the volume of data being assimilated, the maintenance and energy consumption of data centres is not trivial.
