A recent Capgemini study found that 15% of big data initiatives in Europe fail. To ensure your project belongs to the 85% that are successful, I’ve summarized the four major pitfalls to watch out for. (This blog post contains the first two pitfalls, the other two will be published in a different blog post.)
Being aware of these and taking them into consideration will significantly increase the chances of your data analytics project being a success. Don’t worry: you are by no means the only one facing these challenges and pitfalls. In our initial data analytics workshop, we regularly see participants who are encountering them, right through to the end of the project. Here I’d like to share my insights with you from many successful workshops and projects, point out the major pitfalls, and illustrate them with example use cases.
Data analytics and big data are not one and the same – even if they are often used interchangeably.
IT departments often view projects through “big data glasses”. They provide the infrastructure for collecting large amounts of data; for example, in the form of database clusters. These databases store huge volumes of data, which in itself does not create added value for the company. That’s why the data analytics project should always have a clearly defined technological as well as commercial goal. Collecting data just for the sake of it does not bring the company any benefits at all.
Added value only arises when the company leverages the data and the resulting insights. This is where its (non-administrative) departments come in. They define what goals they want to achieve with data analytics – not with big data. They provide the technical understanding that allows data scientists to work with the data in a targeted way. Close cooperation between the ideas provider (department) and the data scientists is therefore an absolute must in order to achieve the defined project goal.
In other words: the success or failure of a data analytics project depends on what and how much technical process understanding is passed on to the data scientists. Data analytics engineers also play an important role here. They support the “translation” and knowledge transfer between the different disciplines. Data analytics engineers draw on their operational experience in manufacturing or logistics and a sound basic understanding of data analytics approaches. The data experts must not only understand the project goal, but also and in particular the correlations in the data. More importantly, they must see its relationship to the real world (machines, sensors, etc.) and the related process steps.
As the Capgemini study shows, IT departments are often the initiators of data analytics projects. This is not in itself a problem, as long as the other departments are closely involved and define the technical objectives of the project.
Before the data scientists can get started, you need to verify the quality and quantity of the data.