Gartner recently reported[i] that nearly 50% of data projects fail on a regular basis. For every successful big data analytics implementation, there are hundreds of projects that start with a bang and end with an empty Hadoop cluster. How can an organization ensure that it is not spending time and money on a project that doesn’t return results?
Successful data projects can often be thought of the same process as panning for gold. The flow of data between and through various departments in a company can be thought of as analogous to water, with important metrics for decision-making as the nuggets of gold that can help consumers make better choices[ii], drive better decision-making[iii] across departments, and even save lives[iv].
As with a real-life mining operation involving large teams and cross-functional communication, the process of "mining" the flow of data can become complicated for a number of reasons. In real gold mining operations, it could be that the pipe through which the water flows was built to incorrect spec, or maybe no one is keeping track of where the water is flowing, and maybe the miners at the end of the line don’t quite know which type of nuggets to keep and which to throw back into the current.
Similarly, there are numerous reasons data projects can fail, from picking the wrong technology, to time constraints, to conflicting management philosophies.
Every project needs someone who is championing the success of the product because they understand its importance to the company, and who can influence management input into the process. Often, executives will perceive the need for better insight and leave it to other teams to plan implementation, moving on to more urgent, tactical business needs. But a big data project involves a potentially large pool of resources to cooperate and shed light on the data: engineers, data scientists, system administrators, data governance and security teams. All of these individuals need to be driven by a single overarching goal, often coming from a sponsor who is close to core company goals and can help offer input to drive the project in the right direction.;