How to Close the Big Data Talent Gap at Your Organisation

Big data offers many benefits for organisations in all industries, but unfortunately, a lot of companies don’t reap these benefits yet. The reason is not that they don’t want to start with big data, nor that they don’t understand what big data is. The challenge many companies face is attracting the right big data talent.

Big data talent is scarce and what is scarce is expensive. Finding the right data professionals for a big data project remains difficult for a lot of organisations. This does not come as a surprise if we look at the figures around global big data talent.

Back in 2011, McKinsey already estimated that in 2018 there would be a shortage of 290.000 data scientists in the United States alone. Globally, demand for data scientists is projected to exceed supply by more than 50 percent by 2018. In the UK, the expected shortage is 56.000 data scientists by 2020. Currently, there are over 500.000 big data jobs listed globally.

As it may be clear, demand is skyrocketing and this will only increase in the future. On the other hand, big data master programs have only started to appear in the past five years or so. Currently, there are only a few hundred data science master programs, barely enough to cater for future demand.

The result is scarcity in available big data talent, which is unfortunately not the only challenge for organisations. Most organisations have no idea who to look for. The term ‘Big Data Scientist’ is over-hyped and almost every organisation wanting to deal, or dealing, with big data is looking to hire this illustrative big data scientist, while they have no clue what the Big Data scientist should actually do! To make matters worse, there are many ‘ordinary’ data analysts out there who have taught themselves a programming language or something else and now call themselves ‘Big Data Scientist’, to make an extra buck of this hype.

These two challenges, therefore, make it difficult for organisations looking to hire big data talent. It is important to know what they really need and which skills they should really hire. This will, on the one hand, prevent hiring ‘fake’ data scientists and, on the other hand, will help you hire a better fit for your business.

Of course, what skills are required all depends on the big data projects being developed, the industry, the type of organisation and many other variables. However, as Louis Colombus writes on Forbes, we can distinguish ten skills that are most in demand by organisations. The top three of this list includes Python programming, Linux expertise and Structured Query Language, with Apache Hadoop currently in position eight.