Many data engineers and analysts find themselves part of a familiar story. They've been hired by a company that's trumpeting an investment in building a "data platform," given access to a patchwork suite of tools and data sets, and then have innovative projects relegated to the low spot on priority lists. How can they break this cycle?
Twenty years ago, you'd be hard-pressed to find a company that employed more than a handful of data analysts -- if that. In 2021, analyzing data is one of the world's hottest professions, with data scientists, engineers, and analysts embedded in nearly every functional area of the modern enterprise, from marketing to customer service, and for good reason.
From Nike's investment in customer data and predictive analytics to centuries-old banking firms such as UBS using AI to detect fraud at scale, business leaders are sold on the concept of disrupting or revolutionizing their legacy industries through data.
Although these companies have found ways to identify use cases and solve specific problems, this model -- embedded data teams solving business-unit-level problems -- isn't always scalable or sustainable.
As more teams across the company become capable of working with data, redundancies creep in, work diffuses, and infrastructure maintenance costs may skyrocket.
We share common challenges faced by companies when building data platforms and highlight a few best practices for overcoming them:
When employees set out to solve a specific problem for their specific team, they likely aren't considering how the solution they build fits into the company's overall ecosystem. Those team-level blinders lead to silos, where different teams doing similar work within the organization don't communicate with or learn from each other.
Working in silos can allow data scientists to move faster, but over time, the silos get in the way of aligning around shared goals and core values about the characteristics of a good data platform. Silos can also lead to bad habits of "empire building" -- employees or teams trying to gain authority or clout by hoarding resources or access to data.
To break down these barriers, start by making sure you understand the full landscape of how data currently operates within your company and who is responsible for those processes. Talk to the data engineers and scientists within core areas and identify possible redundancies, lost efficiencies, and missed opportunities that you can solve with better collaboration.
Start to build a community of practice among these business users. Getting your data analysts and engineers connected now will help streamline their adoption of your future platform.
Just because your CTO uses a buzzword (such as "data-driven") doesn't mean the executive has a clear understanding of how data functions in your organization -- not to mention the data literacy required to understand, create, and transform data assets. Without this knowledge, it's unlikely your CTO will want to invest in your vision of a data platform.
In some cases, leaders may also be reluctant to stop relying on legacy software instead of investing in cloud-based technologies or a more modern data stack.
