If you go by the latest headlines, the data scientist is the most coveted and scarce enterprise commodity on the employment market. But you could just as easily argue that it’s a trendy title that has not nearly lived up to the hype — and could eventually be automated out of a job.
According to a recent McKinsey Global Survey, 86 percent of executives said their organizations have been “at best only somewhat effective in meeting the primary objective of their data and analytics programs,” and one-quarter said they’ve been “ineffective.” Less than four years after being named the “sexiest job of the 21st century,” it’s time to confront some cold, hard realities. Data scientists are both extremely important and set up to fail unless they adapt to a new model for delivering value.
Recently, I was invited by the UC Berkeley School of Information to host a conversation with students and alums on the real-world applications of data science. During the Q&A, my favorite question was whether I thought software like ours at Alpine Data might eventually replace the data scientist all together. My answer at the time was “no,” more or less: Data science will always be something of an art.
You need to intimately understand the problems that can be solved by data science first, which involves a very human process of interacting with the business. Crafting models will always require the subtle translation of real-world phenomena into mathematical expressions. And there is a human element to interpreting and presenting results that would be difficult to automate.
But it’s still true that, over time, more aspects of a data scientist’s work will be done by software. Feature generation has already become less important as models become more sophisticated. Model parameter selection will become increasingly automated — model deployment entirely so. It seems inevitable that the job description is going to evolve.
Consider how the work of the software engineer has changed fundamentally in the last 20 years. They no longer need to write their own logging module or database access layer or UI widget. And agile methods have brought the “customer” more immediately into the development process. More and more, the job of the engineer is to stitch together higher-level components and collaborate with product managers and UX designers.
Similarly, the job of the data scientist will be to take advantage of pre-built components in order to solve a greater variety of business problems.