More than ever, solving the world’s difficult problems requires the power of data. We’re seeing this firsthand as we witness data teams stepping up to mobilize communities during COVID-19.
To best use data, we need strong, collaborative data teams — not just to solve global problems, but to spur innovation. In retail, for example, we can now better tailor and secure shopping experiences; in financial services, we can make smarter, faster decisions that reduce risk; in oil and gas, we can unlock new efficiencies in discovery, extraction, and downstream delivery of energy; and in healthcare, we can leverage predictive modeling and patient tracking to accelerate clinical trials.
The days when data engineers and data scientists worked on different teams are ending. I believe the convergence of data teams will be similar to the convergence of development and operations teams and DevOps in software development – or before that, the rise of full-stack engineering teams. Opportunities abound for data teams to make a massive impact, but success will depend on streamlining efforts and empowering them in the right ways.
First, leaders must acknowledge the negative impact of siloed activity on efficient processes. As this McKinsey article suggests, consider this comparison: A single basketball player can positively impact a team’s performance, but teaching the entire team to work together will lead to better outcomes. Similarly, breaking down silos between data teams and opening space for more collaboration increases efficiency.
Siloed work on data teams can not only create internal friction, but it can also reduce the effectiveness of technology. Take AI as an example: Forty percent of organizations making significant investments in AI don’t report business gains from it, and siloed data teams can further impede harnessing the true power of AI. Siloed teams add friction in iterative model development processes, slowing the development of AI tools. Changes to data pipelines can also take months if they require efforts from multiple disjointed teams. Teams must collaborate cross-functionally to agree on data definitions or metrics and high-quality analytics for AI projects that require combining multiple datasets.
We need to break these silos down, enable collaboration between data engineering and data science teams, and build a new data team structure.
