After watching application teams, security teams and operations teams get the -Ops treatment, data engineering teams are now getting their own process ending in -Ops.
While still in its very early days, data engineers are beginning to embrace DataOps practices.
Gartner defines DataOps as “a collaborative data manager practice, really focused on improving communication, integration, and automation of data flow between managers and consumers of data within an organization,” explained Nick Heudecker, an analyst at Gartner and lead author of Gartner’s Innovation Insight piece on DataOps.
DataOps is first and foremost a people-driven practice, rather than a technology-oriented one. “You cannot buy your way into DataOps,” Heudecker said.
Michele Goetz, a principal analyst at the research firm Forrester, explained that DataOps is like the DevOps version of anything to do with data engineering. “Anything that requires somebody with data expertise from a technical perspective falls into this DataOps category,” she said. “Why we say it’s like a facet of DevOps is because it operates under the same model as continuous development using agile methods, just like DevOps does.”
DataOps aims to eliminate some of the problems caused by miscommunications between developers and stakeholders. Often, when someone in an organization makes a request for a new data set or new report, there is a lack of communication between the person requesting and whoever will follow through on that request. For example, someone may make a request, an engineer will deliver what they believe is what is needed, and when the requesters receives it they are disappointed that it’s not what they asked for, Heudecker explained. This can result in increased frustration and missed deadlines, he explained.
By getting stakeholders involved throughout the process, some of those headaches may be avoided. “[CIOs] really want to figure out how do they get less friction in their companies around data, which everybody’s asking for today,” said Heudecker.
Another potential benefit of DataOps is improved data utilization, Heudecker explained. According to Heudecker, these are some of the questions that organizations may start to ask themselves:
According to Goetz, for companies that have been journeying down the path of “tightening the bolts” of what is needed from a data perspective and how that supports digital and other advanced analytics strategies, it is clear that they need an operating model that allows development around data to fit into their existing solution development track. This enables them to have data experts on the same team as the rest of the DevOps Scrum teams, she explained.
Organizations that are less mature in their data operations tend to still think in terms of executing on data from a data architecture perspective. In addition, a lot of those less mature companies do not handle data in-house, but will outsource it to systems integrators and will take a project-oriented waterfall approach, Goetz explained.
The companies that are already getting DataOps right are typically going to be the ones that already have a DevOps practice in place for their solution development, whether it’s on the application or automation side, Goetz explained. Those more advanced companies also tend to have a model for portfolio management and business architecture that aligns to continuous development. “They’re recognizing there is an opportunity to better fit into the way that you operate around development with those teams so that data doesn’t get left behind and isn’t building up technical debt,” she said.
According to Goetz, this doesn’t just apply to data systems; it encompasses data governance, which traditionally has been the “final bastion of anything anyone wanted to do with the data. It was always playing cleanup,” she said.
