We hear a lot about the potential benefits of big data, but a new study reveals that those benefits are won at a cost of considerable time spent in cleaning up and preparing raw information.
The study by data integration company Xplenty surveyed over 200 business intelligence professionals and finds that a third of them spend 50-90 per cent of their time just cleaning raw data.
Looking at the ‘extract, transform and load’ (ETL) process, including preferences for on-premise or cloud-based solutions, perceived challenges, and the amount of time spent on ETL, the results show that 97 per cent of those surveyed say that ETL is critical for their business intelligence efforts.
More than half (51 per cent) of those polled say that they currently use on-premise ETL solutions. However, 51 per cent of these say that they are ‘strongly considering’ moving all ETL processes to the cloud.
“While many organisations still rely heavily on existing on-premise IT for ETL, the desire to shift to a more cloud-based model has never been stronger,” says Yaniv Mor, CEO & Co-Founder of Xplenty. “Cloud ETL offers a host of benefits over on-premise, from increased agility in resource deployment to reduced costs. As such, the cloud is an increasingly attractive option from both a performance and operational perspective”.
When asked what the biggest challenges were in making data ready for analysis, 55 per cent say integrating data from different platforms, followed by transforming, cleansing and formatting incoming data (39 per cent), integrating relational and non-relational data (32 per cent), and the sheer volume of data that needs to be managed (21 per cent) at any given time.
“BI professionals should be spending the majority of their time evaluating data and deciphering patterns gleaned through the analytics process – not readying data for analytics,” adds Mor. “The more time they spend making raw data analytics usable, the less time they have to generate real value from it. We have to accelerate Big Data’s ‘time-to-insight,’ boosting efficiency and bringing more immediate answers to an organisation so that they can more quickly take advantage of them”.
