As ‘big data’ evolves from buzzword to cutting edge to commonplace, it’s changing the
world around it. Being ‘data-driven’ will no longer be a differentiator but a basic necessity, and without tapping into proprietary and open data, businesses will fail to keep up in a world of savvy start-ups.
Good software won’t cut it. A wide-angled view of customers and their contexts will be crucial. Social media, calendars and connected devices will make up a picture that algorithms will use for predictive analytics and to create personalised services.
For some, that stirs old fears about machines taking human jobs. However, the best companies will combine human talent and tech: if you’re a data scientist, big data shouldn’t make you obsolete anytime soon.
Data scientists are accomplished technical specialists capable of using an array of tools to interrogate data. They answer the questions businesses ask on their data, plus the ones they didn’t even know they should be asking.
Yet, the shortage of data talent is evident. CrowdFlower’s 2016 data science report found that 83 per cent of respondents said there weren’t enough data scientists to meet demand.
Why? Some of the greatest data storage and processing technologies of recent years have come from a small set of the best engineering brains. For example, Hadoop was the brainchild of a handful of engineers at Google but grown and open sourced at Yahoo. Spark was developed at UC Berkeley’s AMPLab.
Although these technologies are huge in the open source tech community, there is a shortage of talent with the analytical experience to understand and deploy these complex technologies effectively.
This makes data scientists expensive: it’s the law of supply and demand. The median salary currently stands at USD 119,000 according to Glassdoor. Good talent comes at a premium, and a great place to work, that also stretches scientists professionally, will be crucial to hiring and keeping people too.
With all these dollars sunk into technical talent and infrastructure, it’s important for companies to get the most out of those investments.