Within most corporate organizations, many (big) data programs can and will fail to create sustained business value. To avoid this risk, and to gain maximum value from data investments, organizations must overcome a number of stumbling blocks within the analytical supply chain — notably a lack of appropriate people, process or technology. Primarily we must focus on breaking down the social silos that impede efforts to adopt agility and build internal partnerships.
Entering Big Data 3.0, where agility has taken the spotlight, the traditional linear paradigm of data management is being challenged to return value. Even with access to large computing power, analysts, DevOps and IT, we know that without full alignment of the business and IT and a focus on breaking down the data silos, outcomes are likely to be less than optimal. In the worst cases, it becomes challenging to the point where project teams may perform their work with as much duct tape as best wishes.
Oftentimes I hear that organizations complete a project only to discover there was a parallel project, sometimes even two other projects, in the pipeline by other teams. This lack of synergy almost predicts that the output will be brittle and fall short of targeted outcomes. From there, moving that process into business value incurs additional technical debt translating it to other groups.
One method to combat this complexity is assuring that the people closest to the data collaborate and take ownership in the project, to guarantee its refinement and utilization, just as a business would do with any precious commodity.
In a perfect world, data and the information it carries for use in the rest of the company should be integrated, protected and shared with the teams who bring a business case for access to the data.